Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Merged by Bors] - feat: Set explicit resources on all containers #359

Closed
wants to merge 30 commits into from
Closed

[Merged by Bors] - feat: Set explicit resources on all containers #359

wants to merge 30 commits into from

Conversation

razvan
Copy link
Member

@razvan razvan commented Jun 2, 2023 •
edited
Loading

@razvan razvan mentioned this pull request Jun 2, 2023
Closed
@razvan
Copy link
Member Author

razvan commented Jun 11, 2023

Tests pass on kind (using the zookeeper operator 0.0.0-pr691):

--- PASS: kuttl (2682.60s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-default_listener-class-external-unstable (176.54s)
        --- PASS: kuttl/harness/cluster-operation_hadoop-latest-3.3.4-stackable0.0.0-dev_zookeeper-latest-3.8.0-stackable0.0.0-dev (232.97s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-default_listener-class-cluster-internal (168.76s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable (240.95s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable (183.74s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal (189.96s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-default_listener-class-external-unstable (150.00s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-default_listener-class-cluster-internal (155.77s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable (156.72s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal (158.22s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-default_listener-class-external-unstable (187.92s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-default_listener-class-cluster-internal (193.21s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable (193.00s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal (200.23s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-default_listener-class-external-unstable (159.74s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-default_listener-class-cluster-internal (159.35s)
        --- PASS: kuttl/harness/smoke_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal (155.08s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-default_listener-class-cluster-internal (239.95s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-default_listener-class-external-unstable (169.09s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-default_listener-class-cluster-internal (136.25s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-default_listener-class-external-unstable (142.66s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal (170.25s)
        --- PASS: kuttl/harness/orphaned-resources_hadoop-latest-3.3.4-stackable0.0.0-dev_zookeeper-latest-3.8.0-stackable0.0.0-dev (154.44s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable (185.50s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable (136.60s)
        --- PASS: kuttl/harness/logging_hadoop-3.3.3-stackable0.0.0-dev_zookeeper-latest-3.8.0-stackable0.0.0-dev (230.04s)
        --- PASS: kuttl/harness/logging_hadoop-3.3.4-stackable0.0.0-dev_zookeeper-latest-3.8.0-stackable0.0.0-dev (261.15s)
        --- PASS: kuttl/harness/logging_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-latest-3.8.0-stackable0.0.0-dev (249.46s)
        --- PASS: kuttl/harness/smoke_hadoop-3.2.2-stackable0.0.0-dev_zookeeper-3.8.0-stackable0.0.0-dev_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal (147.63s)
PASS

@razvan razvan marked this pull request as ready for review June 11, 2023 08:46
@razvan razvan requested a review from a team June 11, 2023 08:46
@maltesander maltesander self-assigned this Jun 27, 2023
razvan
razvan commented Jun 27, 2023
View reviewed changes
Copy link
Member Author

@razvan razvan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

I think we decided that the smoke test shouldn't configure any resources explicitly and just test the defaults. I see we didn't do it for Zookeeper but only for Hive.

Pinging @sbernauer for his opinion.

@maltesander
Copy link
Member

lgtm.

I think we decided that the smoke test shouldn't configure any resources explicitly and just test the defaults. I see we didn't do it for Zookeeper but only for Hive.

Pinging @sbernauer for his opinion.

Im fine with that. But there is no test that explicitly tests resources (e.g. the defaults, explicitly set) etc.

@maltesander
Copy link
Member

maltesander commented Jun 27, 2023 •
edited
Loading

test: https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/hdfs-operator-it-custom/88/

@sbernauer sbernauer self-assigned this Jun 28, 2023
@sbernauer sbernauer changed the title Smoke tests with resource quotas feat: Set explicit resources on all containers Jun 28, 2023
sbernauer
sbernauer requested changes Jun 28, 2023
View reviewed changes
Copy link
Member

@sbernauer sbernauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We must also change the default resources (cpu,memory,pvc) of every role

rust/operator/src/container.rs Show resolved Hide resolved
rust/operator/src/container.rs Show resolved Hide resolved
rust/operator/src/container.rs Outdated Show resolved Hide resolved
tests/templates/kuttl/smoke/20-install-zk.yaml.j2 Outdated Show resolved Hide resolved
tests/templates/kuttl/smoke/30-assert.yaml.j2 Show resolved Hide resolved
sbernauer
sbernauer requested changes Jun 28, 2023
View reviewed changes
Copy link
Member

@sbernauer sbernauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly done

docs/modules/hdfs/pages/usage-guide/resources.adoc Show resolved Hide resolved
rust/operator/src/container.rs Outdated Show resolved Hide resolved
sbernauer
sbernauer approved these changes Jun 28, 2023
View reviewed changes
Copy link
Member

@sbernauer sbernauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx! LGTM when tests pass

@maltesander
Copy link
Member

bors merge

bors bot pushed a commit that referenced this pull request Jun 28, 2023
@razvan @maltesander @sbernauer
feat: Set explicit resources on all containers (#359)
1104fcd 
Part of stackabletech/issues#394

Co-authored-by: Malte Sander <[email protected]>
Co-authored-by: Sebastian Bernauer <[email protected]>
Co-authored-by: Malte Sander <[email protected]>
@bors
Copy link
Contributor

bors bot commented Jun 28, 2023

Pull request successfully merged into main.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

@bors bors bot changed the title feat: Set explicit resources on all containers [Merged by Bors] - feat: Set explicit resources on all containers Jun 28, 2023
@bors bors bot closed this Jun 28, 2023
@bors bors bot deleted the feat/resource-quotas branch June 28, 2023 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@maltesander maltesander maltesander left review comments

@sbernauer sbernauer sbernauer approved these changes

Assignees

@maltesander maltesander

@sbernauer sbernauer

Labels
release/2023-07
Projects
Stackable Engineering
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@razvan @maltesander @sbernauer @lfrancke